Network policy implementation with multiple interfaces

ABSTRACT

The transmission of data on computer networks according to one or more policies is disclosed. A policy may specify, among other things, various parameters which are to be followed when transmitting initiating network traffic. Multiple network interfaces may be installed on a server to enable transmission of data from the single server according a number of discrete configuration settings implicated by the various policies. The multiple network interfaces may correspond to separate physical components, with each component configured independently to implement a feature of a policy. The multiple network interfaces may also correspond to a single physical component that exposes multiple network interfaces, both to the network and to the server on which it is installed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/788,879, filed Feb. 12, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/228,223, filed Dec. 20, 2018, which is acontinuation of U.S. patent application Ser. No. 14/968,625, filed Dec.14, 2015, which is a continuation of U.S. patent application Ser. No.13/536,006, filed Jun. 28, 2012, the entire contents of each of whichare incorporated by reference herein.

BACKGROUND

Generally described, computing devices utilize a communication network,or a series of communication networks, to exchange data. Companies andorganizations operate computer networks that interconnect a number ofcomputing devices to support operations or provide services to thirdparties. Data communicated between the computing networks originatesfrom a network interface of a transmitting computing device and travelsover the computing network. The data is then received at a networkinterface corresponding to a destination computing device. To facilitatethe transmission of the data through the computing network, the data canbe broken down into a series of packets. The serialized packets are thenreceived and re-transmitted by switches and routers until the set ofpackets (e.g., the data) is received at the appropriate destination.

In a simple embodiment, a computing device, such as a server computingdevice, may have a single network interface, typically implemented as anetwork interface card (“NIC”) that receives and transmits data. In morecomplex embodiments, a server may have multiple network interfaces inorder to, for example, increase available bandwidth by adding additionalnetwork cables on which to transmit and receive network communications.For example, a server may support multiple NICs. In another example, aserver may have a single NIC that supports multiple network interfaces,which is generally referred to as a multi-interface NIC. One example ofa multi-interface NIC is a single root I/O virtualization (“SR-IOV”)NIC. SR-IOV NICs typically include a single full-featured hardwarenetwork interface, known as a physical function, and several lightweightnetwork interfaces, known as virtual functions. Together, the physicalfunction and virtual functions are presented as separate networkinterfaces, both to other computing devices and to the operatingsystem(s) executing on the computing device with the SR-IOV NIC.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of various inventive features will now be described withreference to the following drawings. Throughout the drawings, referencenumbers may be re-used to indicate correspondence between referencedelements. The drawings are provided to illustrate example embodimentsdescribed herein and are not intended to limit the scope of thedisclosure.

FIG. 1 is a block diagram of an illustrative network computingenvironment, including a number of computing devices, in which multiplenetwork interfaces may be implemented;

FIG. 2A is a block diagram of several illustrative interactions betweencomputing devices utilizing multiple network interfaces to implementdifferent maximum transmission units;

FIG. 2B is a block diagram of several illustrative interactions betweencomputing devices utilizing multiple network interfaces to implementdifferent bandwidth throttling policies;

FIG. 3 is a flow diagram of an illustrative process for initializing avirtual machine and assigning it multiple independently configurednetwork interfaces;

FIG. 4 is a block diagram of an illustrative computing device hostingvirtual machines and implementing multiple independently configurednetwork interfaces; and

FIG. 5 is a flow diagram of an illustrative process for passing packetsto an appropriate network interface to facilitate policy enforcement.

DETAILED DESCRIPTION

Generally described, the present disclosure relates to implementation ofcomputer networking features. Specifically, aspects of the disclosurerelate to the transmission of data on computer networks according to oneor more policies. Illustratively, a policy may specify a desired networkperformance characteristic, such as high bandwidth or low latency.Policies may be mapped to specific configuration settings, such as amaximum size of network packets to be transmitted, whether interruptcoalescing is to be used, whether to implement bandwidth throttling, andthe like. For example, a server may be configured to transmit datawithin the data center and also over the Internet to externalrecipients. The two different types of traffic may be subject todifferent policies, including maximum packet size and bandwidththrottling.

In some embodiments, multiple network interfaces may be installed on aserver to enable transmission of data from the single server according anumber of discrete configuration settings implicated by the variouspolicies. Illustratively, the multiple network interfaces may correspondto separate physical devices, with each device configured independentlyto implement a feature of a policy. Additional aspects of the disclosurerelate to the use of a single physical component that exposes multiplenetwork interfaces, both to the network and to the server on which it isinstalled. The multiple network interfaces exposed by such a componentmay also be configured to implement various network features in supportof the assigned policies. Each network interface, regardless of whetherit is located on a separate physical device from other networkinterfaces or co-located on a single physical device with other networkinterfaces, may correspond to a network hardware queue. Each networkhardware queue may be associated with one or more corresponding softwarequeues, and the network hardware and software queues may be configured,separately or together, with various settings to implement desirednetwork policies.

Although aspects of the embodiments described in the disclosure willfocus, for the purpose of illustration, on relationships andinteractions between data center components, one skilled in the art willappreciate that the techniques disclosed herein may be applied to anynumber of hardware or software processes or applications. Further,although various aspects of the disclosure will be described with regardto illustrative examples and embodiments, one skilled in the art willappreciate that the disclosed embodiments and examples should not beconstrued as limiting. Various aspects of the disclosure will now bedescribed with regard to certain examples and embodiments, which areintended to illustrate but not limit the disclosure.

With reference to an illustrative embodiment, a server in a data centermay be configured to host a number of virtual machines. Each virtualmachine may be configured to communicate over the data center networkaccording to multiple policies. Polices may specify, among other things,desired network characteristics such as high bandwidth or low latency.The policies may correspond to specific configuration settings or otheroperating parameters, such as the maximum transmission unit (MTU) ofnetwork packets, whether interrupt coalescing is enabled, whether and towhat extent bandwidth throttling is utilized, quality of serviceparameters, and the like. For example, a policy specifying a low latencynetwork may be satisfied, at least in part, by turning off interruptcoalescing, while a policy specifying high bandwidth bay be satisfied,at least in part, by utilizing interrupt coalescing. The specificconfiguration settings used to implement a policy may depend on thespecific hardware and/or software available to the virtual machine. Aspecific configuration setting implemented on one computing device tosatisfy a policy may have no effect or even hinder satisfaction of thesame policy on a different computing device, on the same computingdevice under different operating environment conditions, etc. A resourcemanagement component, such as a hypervisor or operating system, candetermine at the time the virtual machine is launched or at some timethereafter which configuration settings to implement in order to satisfythe desired policy.

One policy implemented with respect to a particular virtual machine orsome set of virtual machines may be satisfied by a network configurationsetting indicating that control plane traffic is to be transmitted at anMTU of 1,500 bytes, while data plane traffic may be transmitted at anMTU of 9,000 bytes. While such a policy may be implemented in software,software implementations are generally inefficient, particularly whencompared offload segmentation provided by hardware network interfaces.Therefore, the server may have multiple network hardware queues, eachwith a different configuration. In the present example, a first hardwarequeue may be configured to transmit data at an MTU of 1,500 byes, and asecond hardware queue may be configured to transmit data at an MTU of9,000 bytes. Accordingly, data plane traffic originating from thevirtual machine may be transmitted via the second hardware queue, whilecontrol plane traffic may be handled by the first hardware queue.

FIG. 1 illustrates an example network computing environment 100 in whichmultiple independently configured network interfaces may be used toimplement or enforce network policies. An independently configurednetwork interface may include a network hardware queue which facilitatestransmission and receipt of data via a medium, such as a network cable.A network hardware queue may have any number of configuration settings,as described in detail herein, some or all of which may be dynamicallyconfigurable. In some embodiments, a network hardware queue may beassociated with a software queue that facilitates dynamic configurationof some or all configuration settings of its corresponding hardwarequeue. Optionally, a software queue may provide dynamic configuration ofa setting that is not configurable or present in its correspondinghardware queue, and vice versa. These various network hardware queuesand software queues, either alone or in combination, may be referred toas independently configured network interfaces or, more generally, asnetwork interfaces.

Network computing environments such as the one illustrated in FIG. 1 maybe implemented in data centers and other environments in which severalcomputing devices 102 communicate with each other over a network 110.For example, an entity may operate one or more data centers whichprovide computing and storage services to internal or external customers122. As described in more detail below, each customer 122 may connect tocomputing devices 102 within the network computing environment 100 toinitiate computing processes, in some cases causing the establishment ofconnections between computing devices 102. While the present disclosurewill focus, for purposes of illustration only, on the operation of anetwork computing environment 100 providing computing services toexternal or internal customers 122 through the use of virtual machines,the systems and processes described herein may apply to anyimplementation of a network computing environment 100, including onewith no separate customer 122 entities or no virtual machine usage.

The network computing environment 100 may include any number ofcomputing devices 102. Each computing device 102 can have one or morenetwork interfaces 104 to facilitate communication with other computingdevices 102 over a network 110. The network 110 may be a local areanetwork (LAN), wide area network (WAN), some other network, or acombination thereof. In addition, the network computing environment 100may connect to another network 120, such as a corporate or universitynetwork, or a collection of networks operated by independent entities,such as the Internet. Customers 122 of the network computing environment100 may communicate with computing devices 102 over the combination ofthe networks 120, 110. In some embodiments, the customers 122 may causea computing device 102 to launch a virtual machine instance to executevarious computing operations for or on behalf of the customer 122. Anynumber of virtual machine instances may be running on a single computingdevice 102 at a given time. In addition, the various virtual machinesrunning on a computing device 102 may be associated with a singlecustomer 122 or with a number of different customers 122.

In some embodiments, the network 110 may be a combination of a substratenetwork on which any number of overlay networks may operate. As will beappreciated by one of skill in the art, network cables and switches mayphysically connect the various computing devices 102 through theirrespective network interfaces 104, and a number of different protocolsmay be implemented to facilitate communication over the physicalinfrastructure (e.g., IP over Ethernet, InfiniB and, etc.). Overlaynetworks may be implemented on top of the substrate network and mayoperate at a higher layer of the network model. Overlay networksfacilitate, among other things, abstraction from the specifics of thesubstrate network. Policies regarding the operation of an overlaynetwork, subnets, the network 110 as a whole, or some other portion ofthe network 110 may be implemented for a variety of reasons, such as toprovide a specific feature or network characteristic, or to guarantee acertain level of service to a customer 122.

Providing varied network services and features concurrently on the samehardware presents several challenges. One challenge, among others, isthat the network interfaces 104 of computing devices 102 may supportonly one configuration at a time, and may not have the capability to bedynamically reconfigured based on the momentary needs of a program,operating system, or virtual machine. As a result, a computing device102 may be limited to providing only those features that are supportedby its network interface 104. Unsupported features may be provided insoftware, such as by modifying the network interface driver or providingthe functionality in the hypervisor or operating system. However,implementing features such as packet segmentation, bandwidth throttling,and the like in software can introduce additional challenges and furtherdegrade performance.

One solution, among others, is to add multiple independently configurednetwork interfaces 104 to a single computing device 102. Advantageously,this allows a variety of features to be made available to the programs,operating systems, and virtual machines executing on the computingdevice 102 while avoiding the inefficiencies and performance degradationof software solutions.

Illustratively, with reference to FIG. 1, a computing device 102 d maycommunicate with computing devices 102 b and 102 c. Computing device 102b may be part of a subnet with a different, lower maximum transmissionunit (MTU) than computing device 102 d, and computing device 102 c maybe part of a subnet with a still lower MTU. Traditionally, the computingdevice 102 d could ensure that its transmitted packets satisfied thediffering MTUs by either (a) utilizing a network interface 104 dconfigured to segment packets to the lowest of the applicable MTUs, (b)segmenting the packets according to the destination subnet MTU insoftware prior to employing the network interface 104 d to transmit thepackets, or (c) transmitting packets which may exceed the MTU of thereceiver's subnet, thereby forcing the intermediary network components,such as routers, or the receiving computing device 102 b, 102 c itselfto segment the packets. As described above, the software solution maycause inefficiencies and performance degradation. In addition, relyingon the receiver or the intermediary network components to enforce theMTU may expose those devices to potential denial of service (DoS)attacks by tying up resources during packet segmentation. By insteadutilizing multiple network interfaces 104 d, 104 e configured totransmit at different MTUs, the computing device 102 d can leverage themore efficient hardware segmentation provided by the network interfaces104 d, 104 e and transmit segmented packets to computing devices 102 b,102 c which do not exceed the corresponding MTUs of the subnets of whichthey are a part.

In another embodiment, one virtual machine executing on a computingdevice 102 d may utilize interrupt coalescing or implement a policy inwhich it is desirable to utilize interrupt coalescing, while anothervirtual machine executing on the same device 102 d may not. Interruptcoalescing allows multiple packets to be received before generating aninterrupt. For example, a received packet may be queued and theinterrupt may be delayed for a length of time (e.g., x microseconds) oruntil a threshold number of subsequent packets arrive (e.g., x packets).In some cases, interrupt coalescing is a discrete configuration settingof the network interface and may not be dynamically switched on or offdepending on the momentary needs of the programs, operating systems, andvirtual machines of the computing device 102 d. Separate networkinterfaces 104 d, 104 e can be provided on a single computing device 102d, with network interface 104 d implementing interrupt coalescing whilenetwork interface 104 e does not. The separately configured networkinterfaces 104 d, 104 e allow the programs, operating systems, virtualmachines, etc. executing on the computing device 102 d to utilize eitherinterrupt coalescing setting concurrently.

Further features may be provided by implementing multiple independentlyconfigured network interfaces, including quality of service guarantees,bandwidth throttling and capping, and the like. The techniques disclosedand claimed are not limited to the example embodiments described herein.

The multiple independently configured interfaces 104 d, 104 eimplemented on computing device 102 d may be separate physical networkinterfaces, such as separate physical network interface cards (“NICs”)each with its own queue for packets. In some embodiments, multiplenetwork interfaces may be implemented within a single physical NIC 104a. For example, single root I/O virtualization (“SR-IOV”) NICs provide anumber of secondary queues in addition to a primary queue of the networkinterface. The primary queue is referred to as a physical function, andthe secondary queues are referred to as virtual functions. Together, thephysical function and virtual functions are presented as separatenetwork interfaces, both to other computing devices and to the programs,operating systems, and virtual machines executing on the computingdevice with the SR-IOV NIC.

As will be described in detail below, each independently configurednetwork interface, whether a separate NIC or a virtual function of asingle SR-IOV NIC, may be assigned to a different virtual machineoperating on a computing device. Therefore, multiple virtual machines ona single computing device may take advantage of discrete sets oftraditionally mutually exclusive features. In some embodiments, one ormore virtual machines may be associated with multiple networkinterfaces. Additionally, in some embodiments the various networkinterfaces may be pooled and made available to each virtual machine asvarious features are needed. The pooling may be managed by a resourcemanagement component, such as a hypervisor or operating system, and maybe transparent to the virtual machines. In further embodiments, acomputing device does not execute virtual machines. In such cases, themultiple network interfaces are made available to the processesexecuting on the computing in a fashion similar to that described hereinwith respect to virtual machines.

Turning now to FIG. 2A, example interactions between computing devices,some of which implement multiple network interfaces, will be describedin greater detail. Specifically, network transmissions of varying sizesbetween computing devices 102 a, 102 b, 102 c, 102 d, which may belocated on subnets with a different MTUs, will be described. Computingdevice 102 a may have an SR-IOV network interface 104 a with two virtualfunctions configured to transmit data across the network at an MTU of1,500 bytes (e.g., standard Ethernet V2 frames) and 9,000 bytes (e.g.,Ethernet jumbo frames). Computing device 102 b may have a single NIC 104b configured to transmit data with an MTU of 4,000 bytes. Computingdevice 102 c may have a single NIC 104 c configured to transmit datawith an MTU of 1,500 bytes. Computing device 102 d may be configuredwith two NICs 104 d, 104 e. NIC 104 d may be configured to transmit datawith an MTU of 9,000 bytes, while the other NIC 104 e may be configuredto transmit data with an MTU of 1,500 bytes.

Communications from computing device 120 a to computing device 102 b or102 c may be transmitted by the 1,500 byte virtual function of theSR-IOV network interface 104 a. If the virtual machine initiating thetransmission prepares a large packet of data for transmission, such as a64 kilobyte packet, segmentation of the packet can be offloaded to avirtual function rather than performing the segmentation in software.The virtual function can efficiently segment the packet into a number ofsmaller packets no larger than 1,500 bytes for transmission to computingdevices 102 b, 102 c. Note that although computing device 102 b islocated on a 4,000 byte subnet, the two virtual functions available tovirtual machines of the computing device 120 a implement segmentation toeither 9,000 bytes or 1,500 bytes. Segmentation into 4,000 byte packetswould be done in software, such as by the SR-IOV NIC 104 a driver, andthen transmitted by the virtual function configured to transmit at amaximum of 9,000 bytes. By utilizing the segmentation offload providedby the virtual function instead of software segmentation, the entiretransmission may be more efficient even though the packets are segmentedinto a larger number of smaller packets.

Communications from computing device 102 b to computing device 102 c maynot take advantage of offloading segmentation to the NIC 104 b. The NIC104 b is configured to transmit packets at an MTU of 4,000 bytes, sosegmentation of larger packets may be offloaded to the NIC 102 c onlyfor segmentation into 4,000 byte packets. However, 4,000 byte packetsare larger than the 1,500 byte MTU subnet of the computing device 102 c.Therefore, the router 202 may segment the packets into a maximum of1,500 bytes before transmitting them to the computing device 102 c.Alternatively, software segmentation may be implemented by the computingdevice 102 b, as described above. Either implementation is not asefficient as segmentation offload provided by a network interface.

Communications from computing device 102 c to computing device 102 d maybe sent to either of the computing device's 102 d two NICs 104 d, 104 e.In order to direct traffic at a particular NIC 104 d or 104 e, the twoNICs 104 d, 104 e may be configured with different media access control(MAC) addresses and different IP addresses (assuming an IP substrate oroverlay network is in use) so that the router 202 can forward thepackets to the appropriate NIC 104 d, 104 e. In some embodiments the twoNICs 104 d, 104 e may be configured with only different MAC addresses,as in the case of an InfiniB and substrate network.

Communications between device 102 a and 102 d may again be transmittedby either network interface to a compatible network interface (packetsfrom the 1,500 byte virtual function of network interface 104 a toeither network interface 104 d, 104 e; packets from network interface104 d to the 9,000 byte virtual function of network interface 104 a,etc.). In addition, both computing devices 102 a, 102 d can initiatedifferent types of network traffic from the independently configurednetwork interfaces. For example, the computing device 102 a may sendcontrol plane traffic to the router 202 from the 1,500 byte MTU virtualfunction, while sending data plane traffic from the 9,000 byte virtualfunction, through the router 202, and finally to the computing device102 d. In another example, a virtual machine operating on computingdevice 102 a on behalf of a customer 122 may communicate with computingdevices of the customer 122 via the internet 120. The computing device102 a can send traffic via the internet 120 from the 1,500 byte MTUvirtual function, while initiating traffic from the 9,000 byte MTUvirtual function to other computing devices of the network computingenvironment 100.

As described above, the various virtual functions of an SR-IOV networkinterface 104 a may be presented as separate network interfaces to thedevices of the network, and to the virtual machines of the computingdevice 102 a itself. In order to direct traffic at a particular virtualfunction of the SR-IOV interface 104 a, each virtual function may beconfigured with different media access control (MAC) addresses anddifferent IP addresses so that the router 202 can forward the packets tothe appropriate virtual function. In some embodiments the two NICs 104d, 104 e may be configured with only different MAC addresses.

FIG. 2B illustrates another network feature with discrete configurationsettings implemented on the same hardware through the use of multiplenetwork interfaces. The interactions illustrated in FIG. 2B result frombandwidth throttling. A single SR-IOV NIC may implement differentbandwidth throttling settings in its various virtual functions. Forexample, the network interface 104 a of computing device 104 a may be anSR-IOV NIC with at least two virtual functions: one with bandwidththrottling implemented at 1 gigabit per second (“Gbit/s”), and anotherat 10 Gbit/s. Note that due to the use of a single physical NIC, thereis only one network cable over which the independently throttled data istransmitted. In some embodiments, multiple physical NICs with differentbandwidth throttling settings may be implemented on a single computingdevice. For example, computing device 102 d may have at least twoseparate physical NICs: one (104 d) with bandwidth throttlingimplemented at 10 Gbit/s, and another (104 e) at 1 Gbit/s.

Turning now to FIG. 3, an example process 300 implemented by a computingdevice for associating one or more independently configured networkinterfaces with a newly launched virtual machine. The process 300 may beexecuted by a computing device 102 or some component thereof, such as anoperating system or hypervisor. Advantageously, the hypervisor mayselect network interfaces based on configuration parameters associatedwith the virtual machine to be launched, thereby providing a set offeatures and services customized to a particular virtual machine eventhough other virtual machines associated with other discrete sets offeatures and services may already be executing on the computing device102 or may subsequently be launched on the computing device 102.

The process 300 begins at block 302. The process 300 may be initiatedupon receipt, by a computing device 102, of a notification, instruction,or other message indicating that a virtual machine instance is to belaunched. For example, a customer 122 may connect to the networkcomputing environment 100 and initiate a computing session. Anotification to launch a virtual machine for the customer 122 may besent to random or selected computing device 102. In some embodiments,each computing device 102 may be configured with network interfacesconfigured with a standard set of configuration settings. In such cases,a computing device 102 may be chosen randomly or based on load balancingconsiderations. In other embodiments, computing devices 102 may not haveidentically configured sets of network interfaces. In such cases, acomputing device 102 may be chosen based on the needs of the customer'svirtual machine.

At block 304, the computing device 102, or some component thereof suchas the hypervisor, may launch a virtual machine for the customer. Thevirtual machine may be launched from a standard virtual machine image oran image customized for or selected by the customer. For example, FIG. 4illustrates an example computing device 102 configured with an SR-IOVNIC 104 having three independently configured virtual functions 404 a,404 b, 404 c and a physical function 404 d. Two virtual machines 402 a,402 b have been launched and are executing on the computing device 102.The hypervisor 400 can launch a third virtual machine 402 c in responseto receiving a notification to do so.

At block 306, the hypervisor 400 may determine the network interfaceconfiguration or configurations to provide for the newly launchedvirtual machine 402 c. The hypervisor may have access to a repository ofconfiguration information to consult when determining the networkconfigurations to provide for virtual machines. In some embodiments,configuration information may be received with the notification tolaunch the virtual machine 402 c, or from the virtual machine instance402 c itself, or from the image from which the virtual machine instance402 c was instantiated. For example, the hypervisor 400 may determinethat the virtual machine 402 c is to be provided with network interfaceswhich implement two separate policies. The hypervisor 400 can thendetermine which network configuration settings, when implemented on theavailable hardware, will satisfy each of the policies. In some cases,the configuration settings applied to two or more network interfaces maybe mutually exclusive, such that a policy is satisfied by one networkinterface or another, but not by both. In the present example, thehypervisor 400 may determine that the two separate policies can besatisfied by two network interfaces: one interface configured totransmit at an MTU of 9,000 bytes and another interface configured totransmit at an MTU of 1,500 bytes. Both interfaces are to have interruptcoalescing disabled.

At block 308, the hypervisor 400 may identify which of the networkinterfaces 404 a, 404 b, 404 c, 404 d to associate with the newlylaunched virtual machine 402 c. Returning to the example above, thehypervisor 400 may identify virtual function 2, 404 b, and the physicalfunction 404 d as providing the configuration settings which are mostlikely, or have a likelihood exceeding a threshold, to satisfy thepolicies associated with the virtual machine 402 c. In some embodiments,the hypervisor 400 or some other component can dynamically configure thenetwork interfaces of the computing device 102 to satisfy the policiesof the virtual machine 402 c. For example, the hypervisor 400 may modifya configuration setting of a network hardware queue or its associatedsoftware queue in order to satisfy one or both policies. The hypervisor400 may determine which configuration settings to identify ordynamically configure based on the current or predicted future operatingenvironment of the virtual machine 402 c, such as the number of othervirtual machines expected to execute concurrently on the same computingdevice 102, the expected network traffic, and the like.

At block 310, the hypervisor 400 may associate the network interfacesidentified in block 308 with the virtual machine 402 c. In someembodiments, this may involve an exclusive assignment wherein the twoidentified network interfaces 404 b, 404 d may only be utilized by theassigned virtual machine 402 c until the occurrence of some triggeringevent, such as the end of execution of the virtual machine 402 c. Inother embodiments the associations may not be exclusive. For example,the hypervisor 400 may provide a pool of network interfaces to multiplevirtual machines, allowing two or more virtual machines to utilize asingle network interface, or similarly two or more VMs may shareutilization of two more network interfaces. After the virtual machineinstance 402 c has been associated with one or more network interfaces,it may transmit and receive data via the associated network interfaces.For example, application software running within the virtual machineinstance 402 c may select which network interface to utilize whentransmitting data based on a policy associated with the data.

Turning now to FIG. 5, an example process 500 implemented by a computingdevice for initiating and receiving transmissions using multiple networkinterfaces will be described. The process 500 may be implemented by acomponent of a computing device 102, such as an operating system orhypervisor 400 as illustrated in FIG. 4, to determine which networkinterface to utilize when transmitting a packet. The process 500 therebyallows the operating systems, virtual machines, applications, and othermodules which may be executing on the computing device to be shieldedfrom specific details regarding network interface assignment andconfiguration. As a result, different network interfaces may be utilizedto transmit and receive data for a virtual machine throughout thelifetime of the virtual machine without any persistent association ofthe virtual machine to any network interface, for example as describedabove with respect to FIG. 3. Alternatively, the hypervisor 400 canforward packets from the virtual machines 402 a, 402 b, 402 c to anappropriate virtual function 404 a, 404 b, 404 c or physical function404 d based on the assignments made by the process 300 described aboveand also based on characteristics of the packet itself, such as its sizeor destination.

The process 500 begins at block 502. At block 504, a component of acomputing device 102, such as a hypervisor 400, receives a data packet.The packet may be received from a program, operating system, virtualmachine, or other module or process executing on the computing device102. For example, a virtual machine 402 c may be executing computingoperations on behalf of a customer. The computing operations may, atsome point, include a network transmission. The virtual machine maycreate a data packet and pass the packet to the hypervisor 400 forforwarding to the appropriate network interface.

At bock 506 the hypervisor 400 determines whether a network policy isapplicable to the transmission of the packet. If a network policy isapplicable, the hypervisor 400 determines the specific policyrequirements to enforce with regard to transmitting the packet.Returning to the example above, the virtual machine 402 c may beassociated with two different MTUs, depending on the type of networktraffic initiated by the virtual machine. Control plane traffic to arouter may be associated with a policy specifying an MTU of 1,500 bytes,while data plane traffic to be transmitted to other computing devices102 within the network computing environment 100 may be associated witha policy specifying an MTU of 9,000 bytes. In the current example, thedata packet may be a large 56 kilobyte packet to be transmitted toanother computing device 102 within the network computing environment100. The hypervisor 400 may therefore determine that the packet is to betransmitted at an MTU of 9,000 bytes.

At block 508 the hypervisor identifies a network interface implementingconfiguration settings satisfying the policy determined to apply to thetransmission of the current packet. In the present example, the virtualmachine 402 c has been associated with virtual functions 404 b andphysical function 404 d. As illustrated in FIG. 4, virtual function 404b is configured to transmit data at an MTU of 9,000 bytes and providescorresponding segmentation offload for that MTU. Physical function 404 dis configured to transmit data at an MTU of 1,500 bytes and providescorresponding segmentation offload for that MTU. Among the two assignednetwork interfaces, only virtual function 404 b satisfies the policydetermined in block 506 above.

At block 510, the hypervisor 400 can forward the packet to the networkinterface identified in block 508. In the present example, the virtualfunction 404 b is associated with the virtual machine 402 c from whichthe packet originated, and it also satisfies the policy under which thepacket is to be transmitted. In response to the determination, thehypervisor 400 can forward the packet to the physical function 404 b.The physical function 404 b segment the large 56 kilobyte packet in anumber of smaller 9,000 byte packets for transmission to the destinationcomputing device 102.

Depending on the embodiment, certain acts, events, or functions of anyof the processes or algorithms described herein can be performed in adifferent sequence, can be added, merged, or left out all together(e.g., not all described operations or events are necessary for thepractice of the algorithm). Moreover, in certain embodiments, operationsor events can be performed concurrently, e.g., through multi-threadedprocessing, interrupt processing, or multiple processors or processorcores or on other parallel architectures, rather than sequentially.

The various illustrative logical blocks, modules, routines, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented inapplication-specific hardware, or in software executed by hardware,depends upon the particular application and design constraints imposedon the overall system. The described functionality can be implemented invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the disclosure.

The steps of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein can be embodieddirectly in hardware, in a software module executed by a processor, orin a combination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of a non-transitorycomputer-readable storage medium. An exemplary storage medium can becoupled to the processor such that the processor can read informationfrom, and write information to, the storage medium. In the alternative,the storage medium can be integral to the processor. The processor andthe storage medium can reside in an ASIC. The ASIC can reside in a userterminal. In the alternative, the processor and the storage medium canreside as discrete components in a user terminal.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

Conjunctive language such as the phrase “at least one of X, Y and Z,”unless specifically stated otherwise, is to be understood with thecontext as used in general to convey that an item, term, etc. may beeither X, Y, or Z, or a combination thereof. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of X, at least one of Y and at least one of Z toeach be present.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it can beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the spirit of the disclosure. As can berecognized, certain embodiments of the inventions described herein canbe embodied within a form that does not provide all of the features andbenefits set forth herein, as some features can be used or practicedseparately from others. The scope of certain inventions disclosed hereinis indicated by the appended claims rather than by the foregoingdescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

1.-31. (canceled)
 32. A system comprising: a network interface devicecoupled to a substrate network on which a plurality of overlay networksoperate, wherein the network interface device comprises a plurality ofnetwork hardware queues, and wherein at least two of the plurality ofnetwork hardware queues are associated with different transmissionparameters; and a processing unit in communication with the networkinterface device and configured to: determine that a software instanceexecuting on the system has initiated transmission of a data packet overa first overlay network of the plurality of overlay networks; select afirst network hardware queue of the plurality of network hardware queuesbased on a first transmission parameter of the first overlay network;and cause the first network hardware queue to transmit at least aportion of the data packet over the first overlay network.
 33. Thesystem of claim 32, wherein the first network hardware queue isconfigured to implement packet segmentation based on the transmissionparameter.
 34. The system of claim 32, wherein the processing unit isfurther configured to determine the transmission parameter based on amaximum transmission unit of the first overlay network.
 35. The systemof claim 32, wherein the processing unit is further configured todetermine the transmission parameter based on a maximum bandwidth of thefirst overlay network.
 36. The system of claim 32, wherein theprocessing unit is further configured to determine an interruptcoalescing configuration of the first network hardware queue, whereinthe first network hardware queue is selected based on the firsttransmission parameter and the interrupt coalescing configuration. 37.The system of claim 32, wherein the processing unit is furtherconfigured to initiate execution of the software instance, wherein thesoftware instance comprises one of a virtual machine instance or anapplication instance.
 38. The system of claim 32, wherein the processingunit is further configured to: determine that the software instance hasinitiated transmission of a second data packet to a second overlaynetwork of the plurality of overlay networks; select a second networkhardware queue of the plurality of network hardware queues based on asecond transmission parameter of the second overlay network, wherein thesecond transmission parameter is different than the first transmissionparameter; and cause transmission of the second data packet via thesecond network hardware queue.
 39. The system of claim 32, wherein tocause the first network hardware queue to transmit at least the portionof the data packet over the first overlay network, the processing unitis configured to cause the first network hardware queue to transmit atleast the portion of the data packet over a data plane network.
 40. Thesystem of claim 32, wherein to cause the first network hardware queue totransmit at least the portion of the data packet over the first overlaynetwork, the processing unit is configured to cause the first networkhardware queue to transmit at least the portion of the data packet overa control plane network.
 41. The system of claim 32, wherein theprocessing unit is further configured to assign the first networkhardware queue and a second network hardware queue to the softwareinstance based on a plurality of network policies associated with thesoftware instance, wherein the first network hardware queue satisfies afirst network policy of the plurality of network policies, and whereinthe second network hardware queue satisfies a second network policy ofthe plurality of network policies.
 42. A computer-implemented methodcomprising: as implemented by a computing system comprising one or moreprocessors configured to execute specific instructions, determining thata software instance executing on the computing system has initiatedtransmission of a data packet to a first overlay network of a pluralityof overlay networks, wherein the computing system comprises a networkinterface device coupled to a substrate network on which the pluralityof overlay networks operate; selecting a first network hardware queue ofa plurality of network hardware queues of the network interface device,wherein the first network hardware queue is selected based on a firsttransmission parameter of the first overlay network, and wherein atleast two of the plurality of network hardware queues are associatedwith different transmission parameters; and causing the first networkhardware queue to transmit at least a portion of the data packet overthe first overlay network.
 43. The computer-implemented method of claim42, further comprising implementing, by the first network hardwarequeue, packet segmentation based on the transmission parameter.
 44. Thecomputer-implemented method of claim 42, further comprising determininga maximum transmission unit of the first overlay network, wherein thefirst network hardware queue being selected based on the firsttransmission parameter of the first overlay network comprises the firstnetwork hardware queue being selected based on the maximum transmissionunit of the first overlay network.
 45. The computer-implemented methodof claim 42, further comprising determining a maximum bandwidth of thefirst overlay network, wherein the first network hardware queue beingselected based on the first transmission parameter of the first overlaynetwork comprises the first network hardware queue being selected basedon the maximum bandwidth of the first overlay network.
 46. Thecomputer-implemented method of claim 42, further comprising determiningan interrupt coalescing configuration of the first network hardwarequeue, wherein selecting the first network hardware queue is furtherbased on the interrupt coalescing configuration.
 47. Thecomputer-implemented method of claim 42, further comprising initiatingexecution of the software instance, wherein executing the softwareinstance comprises executing one of a virtual machine instance or anapplication instance.
 48. The computer-implemented method of claim 42,further comprising: determining that the software instance has initiatedtransmission of a second data packet to a second overlay network of theplurality of overlay networks; selecting a second network hardware queueof the plurality of network hardware queues based on a secondtransmission parameter of the second overlay network, wherein the secondtransmission parameter is different than the first transmissionparameter; and causing transmission of the second data packet via thesecond network hardware queue.
 49. The computer-implemented method ofclaim 42, wherein causing the first network hardware queue to transmitat least the portion of the data packet over the first overlay networkcomprises causing the first network hardware queue to transmit at leastthe portion of the data packet over a data plane network.
 50. Thecomputer-implemented method of claim 42, wherein causing the firstnetwork hardware queue to transmit at least the portion of the datapacket over the first overlay network comprises causing the firstnetwork hardware queue to transmit at least the portion of the datapacket over a control plane network.
 51. The computer-implemented methodof claim 42, further comprising assigning the first network hardwarequeue and a second network hardware queue to the software instance basedon a plurality of network policies associated with the softwareinstance, wherein the first network hardware queue satisfies a firstnetwork policy of the plurality of network policies, and wherein thesecond network hardware queue satisfies a second network policy of theplurality of network policies.